1. Field of the Invention
This invention relates to an image coding apparatus and an image decoding apparatus for use with a system which performs high efficiency coding or decoding of moving pictures to perform efficient transmission or storage of images, and more particularly to an image coding apparatus and an image decoding apparatus which can be applied to processing of, for example, a digital broadcasting system which is performed using a satellite or a ground wave or cable communication network, a digital video disk, a mobile video phone, a PHS video phone or a data base for images.
2. Description of the Prior Art
As a representative one of conventional high efficiency coding systems, the MPEG2 is known which is an international standard system recommended by the ISO/IEC/JTC1/SC29/WG11. For example, xe2x80x9cImage Information Engineering and Broadcasting Techniquesxe2x80x9d, Journal of the Television Engineering Society of Japan, April, 1995 explains the MPEG as a theme of special editing. A coding system of the MPEG2 is disclosed in xe2x80x9c3-2 Video Compressionxe2x80x9d of the same document, pp. 29-60.
The coding system of the MPEG2 is described below.
FIG. 31 is a block diagram showing a basic construction of an ordinary encoder of the MPEG2, and FIG. 32 is a block diagram showing a basic construction of an MPEG2 decoder. Referring to FIGS. 31 and 32, reference numeral 1 denotes a frame re-arranging unit, 2 a subtracting unit, reference characters 3a and 3b denote each an inter(interframe)/intra(intraframe) switching selector, reference numeral 4 denotes a converting unit, 5 a quantizing unit, 6 a reverse quantizing unit, 7 a reverse converting unit, 8 an adder, 9 a first frame memory, 10 a second frame memory, 11 a forward direction motion compensating unit, 12 a bidirection motion compensating unit, 13 a backward direction motion compensating unit, 151 a motion estimating unit, 16 a coding control unit, 17 a variable length coding unit, and 18 a buffer.
Further, reference numeral 100 denotes input image data in the form of digital data, 101 re-arranged input image data, 102 a predictive error image, 103 an original input image or predictive error image, 104 a conversion coefficient, 105 a quantization coefficient, 106 a reverse quantized conversion coefficient, 107 reversed converted image data, 108 a locally decoded image, 109 a reference image from the first frame memory, 110 a reference image from the second frame memory, 111 a forward direction motion predicted image, 112 a bidirection motion predicted image, 113 a backward direction motion predicted image, 115 a determined predicted image, 117 a control signal to the selector, 118 a control signal to the converting unit 4, 119 an adaptive quantization value, 120 a variable length coder, 121 a bit stream, 123 a motion vector, 124 a reference image, and 125 an intra/inter switching signal.
Operation of the conventional image encoder is described below with reference to FIG. 31.
First, an input image signal 100 in the form of a digital signal is inputted to the frame re-arranging unit 1, by which picture frames to be coded are re-arranged.
FIG. 33 illustrates such re-arrangement. Referring to FIG. 33, reference character I denotes an intra (intraframe) coded picture, P an interframe coded picture, and B a bidirectional predictive coded picture. It is to be noted that reference numerals 1 to 10 represent an order in time in which they are displayed.
The first frame is first coded as an I picture, and then the fourth frame is coded as a P picture, whereupon the already coded I picture is used as a reference frame for prediction.
Then, the second frame is coded as a B picture. Thereupon, the I picture of the first frame and the P picture of the fourth frame coded already are used as reference frames for the prediction. In FIG. 33, each arrow mark represents a direction in which prediction is performed.
Thereafter, coding is performed in the construction of I, B, B, P, B, B, P, . . . by similar processing. Accordingly, the action of the frame re-arranging unit 1 is to re-arrange the input image signal 100, in which the picture frames are arranged in order of time, so that they appear in order of coding in order to allow the processing described above.
Subsequently, since predictive coding is not performed for the I picture mentioned above, when the re-arranged image 101 is inputted as it is to the selector 3a, it is transmitted as a selector output 103 to the converting unit 4. On the other hand, for predictive coding for the P picture or the B picture mentioned above, the re-arranged image 101 is subtracted from a predicted image 115 by the subtracting unit 2, and a predictive error image 102 is transmitted as the selector output 103 to the converting unit 4.
Then, the selector output 103 is inputted to the converting unit 4, and a conversion coefficient 104 is outputted from the converting unit 4. The conversion coefficient 104 passes the quantizing unit 5, and a quantization coefficient 105 is obtained from the quantizing unit 5. The quantization coefficient 105 is coded into a variable length code by the variable length coding unit 17, and a variable length coded word 120 is outputted from the variable length coding unit 17.
The quantization coefficient 105 is, on the other hand, inputted to the reverse quantizing unit 6, and a quantization coefficient 106 is outputted from the reverse quantizing unit 6.
Further, the quantization coefficient 106 is reverse converted back to an image level by the reverse converting unit 7, and image data 107 is outputted from the reverse converting unit 7. The image data 107 is, where it is data of the I picture, added to a predicted image 116 selected by the adding unit 8, and a locally decoded image 108 is outputted from the adding unit 8.
It is to be noted that the locally decoded image 108 is written as it is into the first frame memory 9 when it is an I picture, but, when it is a P picture, it is written into the second frame memory 10.
On the other hand, when the locally decoded image 108 is a B picture, it is written into neither the first frame memory 9 nor the second frame memory 10.
Thereafter, when the locally decoded image 108 is a P picture, since it is used only for forward direction prediction, a reference image 124 in the first frame memory 9 is read out, and motion prediction is performed for each macroblock (basic unit for processing of 16 pixelsxc3x9716 lines) by the motion estimating unit 151. The motion estimating unit 151 thus selects one of the macroblocks which has a value nearest to that of the current macroblock as a predicted image, and simultaneously outputs a motion vector 123 therefrom.
The motion vector 123 is inputted to the motion compensating units 11, 12 and 13 surrounded by a dotted line in FIG. 31, and motion predictive pictures are outputted from the motion compensating units 11, 12 and 13.
In this instance, the forward direction motion compensating unit 11 produces a forward direction motion predicted image 111 using a reference image 109 from the first frame memory 9 and outputs a thus determined predicted image 115.
Further, as described hereinabove, the locally decoded images 108 of all macroblocks in a P picture are written into the second frame memory. However, even with the P picture mentioned above, when the macroblocks thereof are intraframe (intra) coded, the frame re-arranged image 101 is outputted directly as the selector output.
Meanwhile, for a B picture, the procedure of coding processing is similar to that for a P picture described above, but different from the processing for a P picture, in that two reference frames are used for prediction.
The motion estimating unit 151 performs forward direction prediction using the reference image 109 from the first frame memory 9, backward direction prediction using a reference image 110 from the second frame memory 10, and bidirection prediction using both of the reference images 109 and 110 to select one of the prediction modes with which a value nearest to that of the current macroblock is obtained, and then outputs a motion vector 123.
In accordance with the thus determined prediction mode, in the motion compensating unit, one of the motion compensating units 11, 12 and 13 which corresponds to the determined prediction mode produces and outputs a predicted picture.
For example, when bidirection motion prediction is selected, the bidirection motion compensating unit 12 produces and outputs a predicted image 115 determined using a bidirection predicted image 112.
After coding of the B pictures of the second and third frames shown in FIG. 33 is completed, image data written in the second frame memory is transferred to the first frame memory. Thereafter, the P picture of the seventh frame is coded, and a decoded picture is written into the second frame memory.
Thereafter, B pictures (fifth and sixth frames) are coded by similar processing to that described above.
When the macroblocks are intraframe (intra) coded, the image 101 after frame re-arrangement is directly outputted as the selector output similarly as in the case of a P picture.
FIG. 32 is a block diagram of a conventional decoder. Referring to FIG. 32, reference character 22 denotes a variable length decoding unit, 107(a) an intra (intraframe) coded picture, and 107(b) a prediction error picture.
Subsequently, operation of the conventional decoder will be described.
A bit stream 121 is stored for a certain period of time into the receiving buffer 18, and a variable length coded word 120 is variable length decoded by the variable length decoding unit 22 and outputted as a quantization coefficient 105.
The processing procedure after this is quite similar to the local decoding processing of the encoder described hereinabove.
When the macroblock is intra decoded, a reverse converted image 107 makes an image 107(a) without passing the adding unit 8, but when the macroblock is inter (interframe) decoded, the reverse converted image data 107 makes an image 107(b). The image 107(b) is added to a predicted image 115 by the adding unit 8, and a decoded image 108 is outputted from the adding unit 8. The decoded image 108 is processed by the displayed frame re-arranging unit 38 such that such decoded images are re-arranged so that they appear in order of time, and finally, an output image 137 is outputted the displayed frame re-arranging unit 38.
The example of a conventional image coder and image decoder described above is a representative apparatus of a type which performs forward direction, bidirection and backward direction prediction coding in combination.
In the example, for coding of a P picture, only forward direction prediction is performed using the first frame memory to perform predictive coding. On the other hand, for coding of a B picture, one of the modes of forward direction prediction, backward direction prediction and bidirection prediction with which a minimum predictive error is provided is selected using the first and second frame memories.
Accordingly, as coding processing proceeds, decoded pictures written in the frame memories are erased. Consequently, for example, even if one of coded pictures processed in the past is similar to a picture of the currently coded frame, since the past decoded pictures have already been erased from the frame memories, the similar coded picture cannot be used for reference, resulting in a problem of lower image processing efficiency.
It is an object of the present invention to provide, in order to solve the problems of the conventional image encoder and image decoder described above, an image coding apparatus and an image decoding apparatus wherein a decoded image obtained in the past can be utilized efficiently as a reference picture and the overall prediction efficiency is improved to achieve moving picture coding and decoding of a high efficiency.
According to an aspect of the present invention, there is provided an image coding apparatus, comprising storage means for storing a plurality of decoded images, motion compensating prediction means for performing motion compensating prediction corresponding to an input image based on the plurality of decoded images stored in the storage means to produce a motion vector and for generating a predicted image based on the motion compensating prediction, prediction error calculation means for calculating a difference between the predicted image generated by the motion compensating prediction means and the input image to calculate a prediction error image, decoding means for generating the decoded images from the prediction error image calculated by the prediction error calculation means and the predicted image, background image storage control means for selecting one of the decoded images which is to be continuously stored in the storage means as a background image and storing the background image into the storage means, and background motion compensation means for performing motion compensating prediction corresponding to the input image based on the background image to generate a motion vector and generating a predicted image based on the motion compensating prediction.
According to another aspect of the present invention, there is provided an image decoding apparatus, comprising storage means for storing a plurality of decoded images, motion compensation means for performing motion compensating prediction based on the decoded images stored in the storage means to generate a motion compensated image, decoding means for generating the coded images from the motion compensated image from the motion compensation means and a prediction error image, background image storage control means for selecting one of the decoded images which is to be continuously stored in the storage means as a background image and storing the background image into the storage means, and background predicted image generation means for generating a background predicted image based on the background image.
The image coding apparatus and the image decoding apparatus of the present invention may be constructed such that the storage means includes a frame memory for storing a decoded image, and another frame memory for storing the background image.
The image coding apparatus and the image decoding apparatus of the present invention may otherwise be constructed such that re-writing of image contents into the storage means by the background image storage control means is performed in units of a picture after a predetermined interval of time or in response to a control signal from the outside.
The image coding apparatus and the image decoding apparatus of the present invention may otherwise be constructed such that re-writing of image contents into the storage means by the background image storage control means is performed in units of a macroblock after a predetermined interval of time or in response to a control signal from the outside.
The image coding apparatus of the present invention may otherwise be constructed such that the background motion compensation means has a variable searching range for a motion vector from the background images.
The image coding apparatus of the present invention may otherwise be constructed such that it further comprises differential vector generation means for holding a motion vector obtained from the motion compensating prediction means or the background motion compensation means and calculating a difference vector between the generated motion vector and the motion vector in the past, and the difference vector is variable length coded.
The image decoding apparatus of the present invention may otherwise be constructed such that it further comprises a motion vector adding unit for holding a motion vector decoded in the past and adding the motion vector decoded in the past to a difference vector to regenerate a motion vector.
According to a further aspect of the present invention, there is provided an image decoding apparatus which outputs a coded bit stream of moving pictures, comprising a plurality of frame memory groups for storing, individually for a plurality of objects which compose a screen, decoded images of the objects in the past, a frame memory selecting unit for selecting, in response to a control signal, into a frame memory of which one of the plurality of frame memory groups a decoded image is to be written, a motion compensation predicting unit for selecting one of forward direction prediction, backward direction prediction, bidirection prediction and background prediction in units of an object using reference images read out from frame memories of the plurality of frame memory groups provided for the individual objects to perform motion compensating prediction, a subtractor for calculating a difference between the predicted image and a current image to calculate a prediction error image, an adding unit for adding the predicted image from the reference images and the prediction error image of the current image, and a variable length coding unit for variable length coding information.
According to a still further aspect of the present invention, there is provided an image decoding apparatus which decodes a coded bit stream of moving pictures, comprising a plurality of frame memory groups for storing, individually for a plurality of objects which construct a screen, decoded images of the objects, a frame memory selecting unit for selecting, in response to a control signal, into a frame memory of which one of the plurality of frame memory groups the coded images are to be written for the individual objects, a variable length decoding unit for variable length decoding the coded bit stream, and a motion compensating unit for selecting one of forward direction prediction, backward direction prediction, bidirection prediction and background prediction in units of an object using reference images read out from frame memories of the plurality of frame memory groups to generate a motion compensated image.
The image coding apparatus or the image decoding apparatus of the present invention may be constructed such that the plurality of frame memory groups include three frame memory groups.
The image coding apparatus of the present invention may otherwise be constructed such that re-writing of image contents of a region in which an object which is a subject of coding is included in the plurality of frame memory groups in which coded images of the object in the past are stored is performed after a certain interval of time or in response to a control signal from the outside.
The image decoding apparatus of the present invention may otherwise be constructed such that re-writing of image contents of a region in which an object which is a subject of decoding is included in the plurality of frame memory groups in which coded images of the object in the past are stored is performed after a certain interval of time or in response to a control signal from the outside.
The image coding apparatus of the present invention may otherwise be constructed such that searching ranges for a motion vector from reference images from the plurality of frame memory groups for the individual objects are variable for the individual objects.
The image coding apparatus of the present invention may otherwise be constructed such that it further comprises differential vector generation means for holding a motion vector in the past obtained by referring to images from the plurality of frame memory groups for the individual objects and calculating difference vectors separately for the individual objects, and the difference vectors are variable length coded.
The image decoding apparatus of the present invention may otherwise be constructed such that it further comprises a motion vector adding unit for holding decoded motion vectors in the past obtained by referring to images in the plurality of frame memory groups for the individual objects for a certain period of time and adding the motion vectors decoded in the past to the decoded difference vectors to regenerate motion vectors for the individual objects.